Rethinking statistical learning theory: learning using statistical invariants

Computational and Statistical Learning Theory

ECE 598MR: Statistical Learning Theory

1. VC classes. Prove the following statements. (a) Let C and C ′ be two classes of subsets of some feature space X. Suppose that C ⊆C ′, meaning that if C ∈C , then C ∈C ′ as well. Prove that V (C ) ≤V (C ′). (b) Let C be a finite class of subsets of X. Prove that V (C ) ≤ log2 |C |. (c) Let X be a finite feature space. For a given k ≤ |X|, consider the class Fk of binary-valued functions f :X→...

Statistical Decision and Learning Theory

This paper reviews and contrasts the basic elements of statistical decision theory [1–4] and statistical learning theory [5–7]. It is not intended to be a comprehensive treatment of either subject, but rather just enough to draw comparisons between the two. Throughout this paper, let X denote the input to a decision-making process and Y denote the correct response or output (e.g., the value of ...

Computational and Statistical Learning Theory

{0, 1}-valued random variables X1, . . . , Xn are drawn independently each from Bernoulli distribution with parameter p = 0.1. Define Pn := P( 1 n ∑n i=1Xi ≤ 0.2). (a) For n = 1 to 30 calculate and plot the below in the same plot (see [1, section 6.1] for definition of Hoeffding and Bernstein inequalities): i. Exact value of Pn (binomial distribution). ii. Normal approximation for Pn. iii. Hoef...

Computational and Statistical Learning Theory

Specifically for p =∞ ( 1 m ∑m i=1 |f(xi)− vi| )1/p is replaced by maxi∈[m] |f(xi)− vi|. Also define Np(F , α, S) := min{|V | : V is an α-cover of (in `p) ofF on sample S} and Np(F , α, n) := sup x1,...,xm Np(F , α, {x1, . . . , xm}) Definition 2. A function F is said to α-shatter a sample S = {x1, . . . , xm} if there exists a sequence of thresholds, θ1, . . . , θm ∈ R such that ∀ξ ∈ {±1}, ∃f ...